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ABSTRACT 


Decision Support Systems have been evolving for over a decade to assist an 
individual or organization in the decision making process. DSS are often difficult to 
justify or evaluate because many of the benefits that they will provide are intangible 
benefits and are difficult to measure. Over the past decade, there have been several 
methods published that can assist in evaluation or justification. 

In this thesis, I propose a framework designed to assist the user in selecting 
the best available method for evaluation or justification for his/her organization. 


The framework considers: 


e The goals and objectives of the DSS and the organization 
e The tangibility of the goals and benefits 


e The need or requirement to quantify goals and benefits 


DSS are continually evolving and are gaining importance in the information 
systems area. Evaluation and justification will become increasingly more impor- 
tant. The framework will provide a basis for selecting the most appropriate method 


available for an organization. 
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I. INTRODUCTION 


A successful DSS can provide an organization the needed advantage when 
trying to compete in an environment that is financially burdened. The problem 
with DSS is: *How does an organization justify spending dollars on a DSS?” 

To justify a DSS, several methods have been presented throughout literature 
over the past decade. I have designed a framework that will assist the user in 
deciding what method is most appropriate for the user's organization. 

This framework is designed to take the user through a process of three steps 
or questions that pertain to the organization's objectives and the purpose of the 
proposed DSS. By following the framework, an organization can select the most 
appropriate method available for not only justifying a DSS, but also for evaluat- 
ing a current DSS. This framework is flexible and can expand to incorporate new 


justification or evaluation methods if and when they become available. 


A. WHAT IS A DECISION SUPPORT SYSTEM 

Decision Support Systems (DSS) are defined as: “A class of information sys- 
tems that draw on transaction processing systems and interacts with other parts of 
the overall information system to support the decision making activities of managers 
and other knowledge workers in organizations." [Sprague and Carlson, 1982] Other 
definitions differ between what an actual DSS is and what the composition of a DSS 
is. Some definitions are very restrictive to the criteria that involve composition of a 
DSS, while others are of a broader scope. 

Restrictive: “DSS are an interactive computer-based system that help decision 


makers utilize data and models to solve unstructured problems.” (Sprague and 


Carlson, 1982] 


Broad: “DSS are any systems that make some contribution to decision mak- 
ing.” [Sprague and Carlson, 1982] 


Both of these types of definitions have drawbacks. The restrictive definition 
greatly limits the DSS category, whereas the broad definition includes any avail- 
able information system. Á viable DSS actually falls somewhere between these two 
definitions. 

Decision Support Systems (DSS) have been evolving for the past decade. The 
purpose of DSS is to provide a decision maker with the resources needed for assis- 
tance in the decision-making process. The information obtained from the DSS lends 
credibility to the final decision, but does not become the final decision. The final 
decision involves the DSS data, as well as other factors, and is arrived at by the 
decision maker. One strength of a DSS is that it can focus on semi-structured and 
unstructured problems that are not normally addressed by traditional information 
systems. 

These points are supported by Hogue and Watson [1983], as listed in their 


criteria for a DSS as: 
e Supports but does not replace decision making. 


Directed toward unstructured and or semi-structured decision 


making tasks. 
e Data and models organized around the decision(s). 


Easy to use software interface. 


Interactive processing. 


TABLE 1.1: DECISION TABLE 


TYPE OF DECISION PERCENT OF RESPONDENTS WHOSE 
PRIMARY USE OF THE DSS IS FOR 
THIS TYPE OF DECISION 


General Long Range Planning 38% 
Strategic Assessment 23% 
Product Strategy 8% 
Negotiations of budgets 8% 
Reporting and Analysis 8% 
Operational Planning and Control 6% 
Acquisition Strategy 2% 
Capital Investment Strategy 2% 
Financial Strategy 2% 
General Budgeting 2% 
Cash Flow Management 2% 


e DSS use and control is determined by the user. 


e Flexible and adaptable to changes in the environment and the decision maker’s 


style. 


e Quick ad hoc DSS building capabilities. 


B. DSS DECISIONS 
A DSS can assist the decision maker with many types of decisions depending 
on the emphasis of the specific DSS. In a survey, [Meador, et al., 1984], a DSS was 


primarily used to support the following types of decisions (See Table 1.1): 


C. DSS BENEFITS 
Numerous studies addressed the benefits that an effective DSS could provide. 


[Hogue, et al., 1985; Keen, 1981; Meador, et al., 1984; Money, et al., 1988]. In one 


survey response, eight top level executives (with little or no DSS experience) listed 


benefits that they perceived to be the most useful: [Alavi, 1982] 
DSS BENEFITS IN DESCENDING ORDER OF USEFULNESS 
e Provide information processing and retrieval capabilities 
e Evaluate alternatives 


Assist in identifying problems 


Assist in interpreting information 


Provide real time analysis of current problems and opportunities 


Suggest decision alternatives 
e Provide ability to ask “what if” questions 
e Manage executive time by scheduling daily activities 


e Increase decision confidence 


This list does not include all of the benefits that can be derived from an effective 
DSS. However, it is important to understand what information top level management 


would like to see when justifying a DSS. 


D. MOTIVATING FACTORS 

What motivates organizations to procure a DSS to assist in the decision making 
process? Benefits are listed in many articles that advocate DSS, but is this enough to 
justify the investment? [Hogue and Watson, 1983] surveyed 18 major corporations 


and listed the motivating factors which led to the development of DSS. 


e Accurate [nformation was cited by 12 corporations. 


e Organizational Champion was cited by 8 corporations. 


New Information was cited by 6 corporations. 


Managerial Mandate was cited by 4 corporations. 


Timely Information was cited by 3 corporations. 


e Cost Reduction was cited by 1 corporation. 


While costs and tangible benefits are crucial to an organization, these findings 
on motivation show how important intangible benefits have become in the DSS 


arena. 


E. WHY DO ORGANIZATIONS NEED DSS? 

DSS can provide numerous benefits ranging from increased decision alterna- 
tives to improved communication. DSS that provide increased decision alternatives 
could save (or earn) thousands or even millions of dollars simply by providing the 
decision maker with possible alternatives which may have been overlooked. This is 
Just one example of a potential benefit. DSS can also provide sensitivity analysis 
functions to assist a decision maker in choosing between alternatives. The list of 


benefits is long and the value of a successful DSS could be priceless. 


F. WHY DO ORGANIZATIONS NEED TO EVALUATE THEIR DSS? 

While several DSS have become successful, others have been failures. DSS 
are only as good as their input. While it is easy to pick up a magazine and read 
about a successful DSS, it is not as easy to find articles on DSS that were failures. 
Some organizations invest in DSS which ultimately fail. Thus, there must be some 


method to evaluate the success of an organization's DSS. 
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Evaluation and feedback are very important to any operation. Before an or- 
ganization implements a DSS, an evaluation process should be in place. This can 
help determine if the outcome is worth the investment. Evaluations should be done 
on a continuing basis and then the combinations of the evaluations could be used 


to help justify upgrade a DSS when the time comes to do so. 


G. SCENARIO 
The Commanding Officer for the US Army Recruiting Command knows 


that a decision support system can help decision making tasks such as: 
e From where are the majority of recruits for the army coming? 
e What can be done to increase recruiting in each geographic region? 


e What effect will increased advertising have in certain regions? 


How much should be spent on advertising? 


The Commander would like to procure a DSS but must justify the expenditure 
to his supervisors. Goals for the DSS are to increase recruitment and to increase 
the overall quality of the individuals that the U.S. Army recruits. 

Analysis to determine proper method of justification for this DSS will be pre- 
sented in Chapter IV. 


H. SUMMARY 

Many articles have been written about the methods used to justify or evaluate 
decision support systems. 

Hogue and Watson's survey [1983] of 18 corporations also lists the methods 
used by the participants to justify building the DSS (See Table 1.2): 


These and other justification methods will be examined in Chapter II. 
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TABLE 1.2: JUSTIFICATION OF DSS 


Justification Method Percent of Corporations 
Intuition 44% 
Value Analysis 33% 
Cost and Hard Benefits 17% 
Cost and Intuitive Benefits 6% 


A notation in the Hogue and Watson survey states, *Ninety-Four percent of 
the corporations surveyed made no formal attempt to measure the financial impact 
of the DSS after it became operational." 

The purpose of this thesis is to survey the existing methods for DSS justifi- 
cation and evaluation and to present a framework to assist the decision maker to 


decide the most appropriate method available to achieve the corporate needs. 


II. JUSTIFICATION 


A. INTRODUCTION 

The purpose of this chapter is to provide information on justification methods 
that are available. Justification allows for the prediction of future requirements, 
benefits and costs to evaluate the procurement of a DSS. Several methods of justifi- 
cation of DSS have been published in the literature. The discussion of each method 
is not intended to be a rewrite of the original author's work. Instead, its purpose 
is to give a user some background information on each method and to provide the 


user with the original author's work for further elaboration. 
g 


B. JUSTIFICATION METHODS 


The following methods are described in the DSS literature: 
e Excess Tangible Costs 
e Value Analysis 


Benefit Profile Chart 


e Incremental Analysis 

e Aggregate Benefit Value 
e Profit Index 

e Savings/Investment Ratio 
e Intuition 


e Cost Benefit Ratio 


1. Excess Tangible Cost 


The Excess Tangible Costs method [Litecky, 1981] is used to quantify 


intangible benefits. The technique itselfis simple, but there are assumptions which 


should be met before using it: 


ho 


. Tangible costs and benefits are relatively easy to estimate. 
. Intangible benefits are most difficult to estimate. 
. Tangible costs are ordinarily much greater than tangible benefits. 


. Intangible costs are insignificant. 


The steps involved in this method are: 


Step 1. Summarize all of the tangible costs (i.e., program development, in- 


vestment in equipment, etc.). 


Step 2. Analyze the sum of all tangible benefits (1.e.; cost displacement bene- 


fits, decreased operating costs, etc.). 


Step 3. Subtract the tangible benefits from the tangible costs. 


This, according to the assumptions, will result in “Excess Tangible Costs”. 
The Excess Tangible Costs is an important figure to management because it 


is the amount which must be overcome by the intangible benefits. 
Step 4. List the intangible benefits. 


Step 5. Derive a discount factor for each intangible benefit. The discount 
factor should reflect the probability that the particular benefit will result in 


overcoming one tangible dollar. 


e Step 6. Multiply the discount factor (probability) by the Estimated Monetary 
Benefit (EMB). This EMB is the amount the benefit would achieve if there 


was 10096 probability of achieving success. 


Ihe EMB and the discount factors are derived via a feasibility study. 


Example 
Tangible Costs ($25,000) 
- Tangible Benefits 5,000 
Excess Tangible Costs ($20,000) 


Intangible Benefits 


Benefit #1(10,000) x Discount Factor( .3) $3,000 
Benefit #2(10,000) x DF (.5) $5,000 
Benefit #3(20,000) x DF (.4) $8,000 
Benefit #4(25,000) x DF (.3) $7,500 
Total Intangible Benefits $23,500 
NET $3,500 


Therefore, a combination of tangible and intangible benefits will offset the tangible 
costs by a net amount of $3,500. 
The term Excess Tangible Costs relates to the dollar amount that must 
be overcome by the intangible benefits. 
2. Value Analysis 
The Value Analysis method was first introduced by Keen [1981] and was 


divided into eleven steps by Smith [1983]. The eleven steps are: 
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Step 1. Define the benefits to be obtained if a prototype system is developed. 
Step 2. Determine the amount users are willing to pay to obtain the benefits. 
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Step 3. Determine whether a prototype can be implemented within the “cost 


threshold” established by the user. 
Step 4. Design the prototype and measure its use and cost. 
Step 5. Review and extend benefits list if necessary. 


Step 6. Define computer hardware and software which would permit a more 


complete system to be evolved from the prototype. 
Step 7. Determine cost of expanding the system. 


Step 8. Ask users to determine level of benefits which must be obtained to 


justify investing in an expanded system. 


Step 9. Assuming users provide a feasible new “cost threshold”, design a 


second version of the system. 


Step 10. After the second version is implemented, measure its use and costs 


and determine new cost threshold for a possible third stage of development. 


Step 11. Continue steps 5-10 until users and systems designers are satisfied 


that the best solution has been achieved under existing constraints. 


This method allows the users to determine what benefits are important 


and place a monetary figure to each benefit. The monetary values are based on 


what the user be willing to pay to obtain a specific benefit? This approach uses 


the “quick hit” approach when developing a DSS and utilizes prototypes to get a 


working model in the users’ hands as quickly as possible and at a low initial cost. 
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3. Incremental Analysis 
Incremental Analysis [Smith, 1983] methodology forecasts the probable 
need of a DSS or Decision Tool, and then determines the various ways to accomplish 
these items without building or buying the DSS. “The first step in using incremental 
analysis is to predict future workloads and to assign probabilities to the forecasted 
workload. Systems personnel then determine alternative methods for accomplishing 


the workload and assign costs to each alternative.” [Smith, 1983] 


Example 

A marketing firm believes there is a 7076 chance that they will need a DSS 
in the next two years to assist in determining a marketing strategy for geographic 
regions and advertising dollars needed to promote the strategy. 

The alternative is that there is a 30% chance that a DSS will not be 
needed. 
Problem: Determine the alternative ways to accomplish the same goal and deter- 


mine costs for each. 


1. Hire additional personnel to determine marketing strategy. 


2. Subcontract to an outside bureau. 


DSS Needed DSS Not Needed Total Expected 


(.7) (.3) Cost 
Hire Marketing $150,000/year 0 $105,000 
Personnel 
Using Outside — $100,000/year 0 $ 70,000 
Consultants 
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From this analysis, the least costly alternative is to hire an outside con- 
sultant at $70,000 per year. If an appropriate DSS be acquired for less than $70,000 
per year, the firm should explore either buying or building a DSS. 

4. Profitability Index 

The Profitability Index method is simply an accounting exercise used to 
determine a project's worth. The profitability index is a ratio that accounts for cash 
inflows into a project and the amount of investment required. [Garrison 1988] It 
takes into account only the tangible costs and benefits that occur with a project. In 
most cases, DSS provide intangible benefits. This method would be inefficient if an 
organization is interested in any intangible benefits. 


The formula for profitablity index ıs: 


Present Value of Cash Inflows 


Investment Required 


The higher the profitability index, the more desirable the project. The 
profitablity index it allows for more than one project to be evaluated against other 
projects that may or may not provide the same service. 


Project A 


. . $250,000 Est. cash inflow 
e E lo A 
Hire Marketing Personnel ooo pestes 


Required 


Profitability Index = 1.67 
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Project B 


. $150,000 Est. cash inflows 
Purchase of a Marketing DSS — 50000 ent — 
Required 


Profitability Index = 2.14 


Using this simple example shows how Project B has the higher Profitabil- 
ity Index, thus, it should be selected over Project A. 
5. Benefit Cost Ratio 
The Benefit Cost Ratio [Naval Data Automation Command (NDAC), 
1980] can be used when costs and benefits are unequal. To determine the economic 
desirability of an investment, the benefits are divided by the costs, providing a single 
figure for comparison. The value represents the amount of benefits obtained per unit 


of cost. 


a : 
a BCR = Quantifiable Output Measure 


Uniform Annual Cost 


e (UAC) Uniform Annual Cost = Present Value 
N TOM 


e (QOM) Quantifiable Output Measures = Tangible Benefit 


[Economic Analysis Procedures for ADP, NDAC, 1980| 
The terms by and by are factors taken from Appendix B. They are determined by 
the project life. 
Example 

Investment decisions occur randomly over the course of the year. The 
present manual system for evaluating an investment decision takes approximately 3 


days (24 man-hours). An Investment DSS can analyze investment decisions more 
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rapidly, allowing a manager to complete the process in one day (8 man-hours). 
Over the year many investment decisions are overlooked, due to the inability of the 
investment department to keep pace. 

QOM manual = 121 Investment Analysis Opportunities per year 

QOM DSS = 365 Opportunities per year 


Cost Category Manual DSS 


One-time (year one) 0 $50,000 (DSS) 


Recurring (year 2-9) $50,000 (salary) $50,000 (salary) 


50,000(6.042 — .954) 254, 400 


al osa 5088 


= $50,000 





50, 000(.954) + 50, 000(6.042 — .954) 302, 100 





Ios 6.042 — .954 Tau 
QOM 121 ` 
BCR manual = UAC 50,000 = .00242 
QOM 365 
BCR DSS = 7,7 En 006147 


The proposed DSS has a higher BCR than the current manual system. 


Therefore, it is the more cost effective alternative. 
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6. Aggregate Benefit Value 
The Aggregate Benefit Value (ABV) method [NDAC, 1980] is one of many 
weighted methods that are available. Weighted methods may be more applicable in 
the DSS arena due to the fact that many of the benefits are intangible. ABV works 
similarly to the Cost Benefit Ratio, method discussed above. 
The example for the ABV method uses the same UAC figures derived 


from the previous example, with the major difference that no QOM can be derived. 


Example: 

The vice-president wants you to evaluate a DSS and compare it to your 
present manual system. His major concerns are the need for a faster response 
time to unexpected investment opportunities and the ability to answer “what if” 
scenarios. Other, less important factors include better use of resources and more 


effective teamwork. (See Tables 2.1 and 2.2) 


Using this method, the DSS again appears more efficient because it has a higher 
BCR. 
* The factor weights and the rankings are determined prior to the evaluation. The 
factor weights are determined by the user depending on the value the user has placed 
on each decision factor. The rankings are then determined by system personnel, 
normally on a scale of 1 to 10. (Faster response time is three times higher for the 
DSS as compare to the manual method, thus the rankings of 9 for the DSS and only 
3 for the manual method.) 

The Aggregate Benefit Ratio allows a user to factor in the intangible 
benefits, although they are difficult to quantify. However, caution must be exercised 


to prevent the introduction of personal opinions and biases. There always exists the 
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TABLE 2.1: BENEFIT RANKING DSS 


Decision Factor *Factor Weight *Ranking Product 
Faster Response Time 3 9 27 
Ability for “What If” scenarios 3 7 al 
Better Use of Resources 2 8 14 
Teamwork 2 6 > 
TOTAL 74 
UAC (59,375 as determined in CBR example) 9.9375 
BCR 12.46 

TABLE 2.2: BENEFIT RANKING MANUAL 

Decision Factor *Factor Weight *Ranking Product 
Faster Response Time 3 3 9 
Ability for “What If” scenarios 3 5 15 
Better Use of Resources 2 9 10 
Teamwork 2 5 10 
TOTAL 44 
UAC (50,000 as determined in CBR example) 5.0000 
BCR 8.8 


E 


possibility of adjusting factor weight or rankings as needed. Adjustments will bring 
about a different conclusion. 
7. Savings/Investment Ratio - (SIR) 

A Savings/Investment Ratio (SIR) [NDAC, 1980] is the relationship be- 
tween future cost savings and the investment necessary to obtain those savings. A 
SIR of 1 indicates that the present value of the savings is equal to the present value 
of the investment. For an investment to be economically sound, the SIR value must 
be greater than one. SIR is a characteristic of cost only and does not consider 
benefits (except cost savings). 


SIR is used when evaluating an alternative against the present situation. 


Example: 

Presently you are the head of an investment decision department that 
consists of four employees and four personal computers. The department is respon- 
sible for analyzing investment opportunities for a local bank. The four employees are 
paid $40,000 each per year as a salary. The bank has the opportunity to purchase 
an Investment DSS for $50,000. The Investment DSS would decrease the workload 
of your department by 25%, allowing you to reassign one of your four employees to 
a different department and saving $40,000 per year in salaries that you would have 
had to pay to hire another investment analyst. The DSS has a two year service 


guarantee but possibly could be obsolete after 2 years. 


PVs — $40,000 (1.821) = $72, 840 
PV, $50,000 * ~ $50,000 





«SIR= — 1.4568 


Since the SIR is greater than 1.0, the investment is economically sound. 


The present value of the DSS is greater than the present value of the cost. 
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PVS is the present value of the savings. (All future savings must be 
discounted by using factors from Appendix B to bring the value to a present day 
figure.) 

PVlis the present value of the investment. (If the investment is a one time 
investment that 1s paid in full there is no need to discount the amount. However, if 
the investment occurs over a specified period of time the the factors from Appendix 
B should be applied.) 

* Project Year Discount Factors such as 1.821 can be found in Appendix B. 
8. Benefit Profile Chart 

The Benefit Profile Chart method |Smith, 1983] uses a comprehensive 
checklist of benefits (See Appendix A) used by system professionals to help in 
preparing justification documents. The checklist can be adapted to any organı- 
zation's needs. By graphically displaying the benefit charts, a system professional 
can better justify a DSS package to top executives. Àn example of a Benefit Profile 
Chart can be found in Appendix A. 

9. Intuition 

One of the most-used strategies is simply the use of intuition. {Hogue 
and Watson, 1983] Watson found through an empirical study that 44% of the re- 
spondents to his questionnaire used intuition over any other method when justifying 
procurement of a DSS. 

Also along these lines is the motivation to procure a DSS because the 
competition has employed one. Many organizations are motivated by successful 
decision support systems that have been employed by like organizations. The greater 
degree of success from an organization’s competitors, the larger the motivation factor 
for procuring a DSS. In Hogue and Watson’s survey [1983], organizational champion 


was cited by 44% of the corporations as a motivating factor for procuring a DSS. 
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III. EVALUATION 


A. INTRODUCTION 

A current operating DSS should be evaluated with respect to its effectiveness, 
costs, and benefits. This chapter provides information on the evaluation methods 
that are available. The discussion of each method is not intended to be a rewrite of 
the original author's work. Its purpose is to give a user some background information 


on the method and reference to the original author's work for further investigation. 


B. EVALUATION METHODS 


The following methods are described in the DSS Literature: 


e Return on Investment (ROI) 

e Expected Value 

e Value Analysis 

e ROI/w Value Analysis or User Satisfaction 


Profit 


e User Satisfaction 


1. Return on Investment 
The Savings/Investment Ratio [Garrison, 1988] can be used to calculate 
the actual return on investment. The savings/investment ratio is the relationship 
between cost savings and the initial investment with any operating/maintenance 


costs, to obtain those savings. A SIR of 1 indicates that the present value of the 
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savings is equal to the present value of the investment. For an investment to be 
economically sound, the SIR must be greater than 1. The SIR ratio is a characteristic 
of cost only and does not consider benefits (except cost savings). 

SIR is used when evaluating a new system against the current one. 

Example: 

The investment decision department invested in a DSS three years ago. 
The chief of the department would like to determine if the investment was profitable. 
The DSS the department utilizes had allowed them to eliminate one of the paid 
analyst postions. The cost savings from saved salaries is $40,000 for the first year 
and 42,000 and 43,000 in the second and third years. In the fourth year the DSS 
became obsolete because of several changes in the surrounding environment. The 
department was then forced to hire another analyst. The DSS had provided several 
intangible benefits, including increased alternative choices and faster response time. 
The initial investment was $50,000 with maintenance and upgrade costs of $15,000 
over the four years. 


Savings  $125,000 


iim Investment 865,000 


END 


Since the SIR is greater than 1.0, the DSS was a sound investment. 

This DSS returned nearly twice the investment. This is using only tan- 
gible costs. The fact that there were also associated intangible benefits makes it 
even more successful. The fact that the DSS was a sound investment although 
it eventually became obsolete, could be sufficient justification for investigating the 
procurement of a new DSS. 

Return on Investment could also be calculated using the formula: [Polokoff, 


1990] 


Benefits — Depreciation 


Odes Costs 
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The DSS in the above example had a 4-year expected life (for deprecia- 
tion purposes), but in the fourth year, the department hired another analyst, thus 


providing no cost savings for the fourth year. The the figures would show: 


Benefits = Il — $31,250 
Depreciation — c — $12,500 
Costs — $65,000 (50,000 initial + 15,000 maintenance) 
ROI per year — 9L 7712399 = .2885.X 100% = 28.85% 


ROI = 28.85% 


The organization's standards would dictate if the ROI was an acceptable 
amount. 

Other accounting methods are also available for use in determining prof- 
itability of a project. The major point to keep in mind with these methods is that 
intangible benefits are overlooked. 

2. Profit 

Profit (Garrison, 1988] is an area that can be used to evaluate the success 
of a DSS, but only after an extended period of time. The DSS could operate for 
several years before anyone would be able to attribute the change in profit level to 
the success of the DSS. 

Example 1: 

A successful organization (without the aid of a DSS) enjoys a profit of 
one million dollars per year over three years. A DSS is then built to help them pick 
better investment opportunities. The first year profits increase to 1.2 million. The 
second and third year profits increase to 1.3 and 1.4 million, respectively. 

It could be assumed that the organization appears to be turning a bigger 


profit due to the use of the DSS. However, a part of the increase could be due to 
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many other factors which would have brought about the increase without the DSS's 
involvement. The true test then would be to operate without the use of the DSS 
and to compare profit levels. Most organizations would not attempt this strategy. 
The best way to evaluate a DSS using this method would entail a thorough analysis 
of all the environmental factors that could possibly lead to increasing or decreasing 
profits. 

Example 2: 

A successful organization (without the aid of a DSS) enjoys a profit of 
one million dollars per year over three years. 

They then build a DSS to help them pick better investment opportunities. 
The first year their profits decrease to .9 million. The second and third year profits 
decrease to .8 and .7 million, respectively. 

The organization now appears to be losing profits due to the DSS. 

This situation is easier to remedy. If the DSS was scrapped or rebuilt, 
several more years would need to pass to check the profit levels to see if an increase 
results. 

However, environmental factors could be causing the decreased profit. In 
this example, it is easier to doubt the effectiveness of the DSS. Whether or not the 
profits increase or decrease, it would take a long time to determine whether the DSS 
has been successful using this method. 

3. Expected Value 

The accurate use of this method [Smith, 1983] requires an experienced 

and knowledgeable person in the field of statistics, economics, systems analysis and 


DSS to follow these steps: 
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e Step 1: List the DSS benefits and assign them to a group of possible cost 


reduction benefits (Group A, 8, C. . .column 1). 


Step 2 Assign a possible cost reduction range (column 2) and a probability 


of occurrence (column 3) to each group. 


e Step 3: Multiplying the midpoint of the assigned range (column 4) times the 


total cost of the project (assign the product to column 5). 


e Step 4: Then multiply the probability of occurrence times (column 5) to arrive 


at the probable savings that is listed in (column 6). 


Example 


Group Benefit 


Group A_ Increased number of alternatives 
New insight and learning 
Improved communications 
Control 


Group B Make better use of data 
Better understanding of business 


Group C Ability to carry out ad-hoc analysis 
Fast response to unexpected situations 
Better decisions 


The actual cost measured in dollars from this example was $1,000,000. 
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l 2 3 4 5 6 


Grp Poss. Cost Probability Midpoint Cost Probable 
Reduction of Reduction Savings 

Occurrence C4 x Total Cost C3x C5 

A O-1096 70% 5.0% $50,000 $35,000 

B 10-15% 50% 12.5% $125,000 $62,500 

C 15-25% 40% 20.0% $200,000 $80,000 

Total Probable Savings $177,500 


Using this method, it is shown that of the $1,000,000 in actual costs, 

$177,500 is the probable savings from the intangible benefits alone. 
4. Value Analysis 

Value Analysis [Keen, 1981] was discussed in Chapter II. However, value 
analysis can also be used to evaluate a DSS. Once a DSS has been up and running 
in an organization, the question may exist whether to update or scrap the DSS. 
Value Analysis allows the evaluator to determine what the user is willing to pay in 
order to keep his present DSS and to determine if there are other benefits that can 
be added to the DSS. 

Steps 8 through 11 allow the organization to effectively evaluate their 
DSS by going back to the users and discussing expansion or scraping the present 


DSS. 


e Step 8 Ask the users to determine level of benefits that must be obtained to 


justify investing in an expanded system. 


e Step 9 Assuming users provide a feasible new “cost threshold”, design a second 


version of the system. 
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e Step 10 After the second version is implemented, measure its use and costs 


and determine new cost threshold for a possible third stage of development. 


e Step 11 Repeat steps 5-10 until users and systems designers are satisfied that 


the best solution has been achieved under existing constraints. 


The evolving DSS that Value Analysis provides is an excellent way of 
continually evaluating an organizations DSS. If users are pleased with the initial 
versions of the DSS, the organization can continually update the DSS to include 
more functions. 

5. User Satisfaction 

Several methods are available to measure user satisfaction. [Baroudi and 
Orlikowski, 1988] However, a common method is the use of a questionnaire-type 
survey. Surveys can help the organization determine the features of a DSS which 
are the most beneficial, used, or successful and ıt can also help determine those 
features that are worthless or dissatisfying. There are cases where DSS tools and 
user satisfaction surveys have not produced consistent results. These studies showed 
that user satisfaction surveys can sometimes be deceiving. In an assessment [Aldag 
and Power, 1986] of computer-assisted decision analysis, users thought favorab. of 
the decision tool, however using the decision tool did not improve their decisions. In 
a separate study [Gallupe and DeSanctis, 1988], the decision tool greatly enhanced 
the quality of the decision, however, satisfaction and confidence were lower than the 
group that did not use the decision tool. Despite these findings, user satisfaction 
surveys are still widely used and accepted. Most surveys use a Likert-type scale, 
allowing the respondent to evaluate on a scale of 1 to 7, the features of the DSS. Ap- 
pendix C shows an example of a User Satisfaction Survey. [Baroudi and Orlikowski, 


1988] 
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6. ROI w/Value Analysis or User Satisfaction 

This method combines two methods [Baroudi and Orlikowski, 1988] [Gar- 
rison, 1988] and attempts to balance the tangible benefit side (ROI) with the intan- 
gible benefits (Value Analysis or User Satisfaction). 

The methods have been described above. An organization could run 
both methods to determine the bottom line return on their investment and to also 
give consideration to intangibles by implementing value analysis or user satisfaction 
techniques. By implementation of this method, the organization would evaluate 
their present DSS and would also provide data for future justification to updated 
or modify the current DSS. By implementing value analysis or user satisfaction 
techniques, it will be easier to overcome a lower than expected ROI (provided that 
the user satisfaction level was high with the initial DSS). 

This method help to satisfy those who want to see a bottom line figure, 
while at the same time taking into account the intangible benefits that are always 


present in the DSS arena. 
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IV. A FRAMEWORK FOR DSS 
EVALUATION AND JUSTIFICATION 


A. INTRODUCTION 

There are several ways to evaluate or justify an ınvestment. Procurement of 
any system, whether a DSS or a traditional information system, should be thought 
of as an investment. This chapter presents a framework that will assıst a user in 
selecting an appropriate method, or set of methods, to evaluate or justify a DSS. 
The framework that I have designed is a multi-level decision tree. (Figure 4.1) 

The decision tree’s purpose is to guide the user through 3 levels of questions 
to arrive at a methodology which will assist his/her organization when evaluating 
or justifying a DSS. The questions within the decision tree are designed to separate 
the available methods. The answers to the questions within the decision tree should 
be based on the user's organizational goals. Upon answering the questions, the user 


will arrive at a methodology which will best fit organizational needs. 
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Figure 4.1: Framework For Evaluating or Justifying DSS 


B. LEVEL 1 

The first level of the framework addresses the purpose of the DSS assessment. 
[s the user trying to justify future procurement (buy or develop) a DSS, or is he 
looking to evaluate the current DSS in his organization? Several methods are avail- 
able to justify and to evaluate a DSS, however, most methods apply only to either 


justification or evaluation. 


C. LEVEL 2 

The second level in this framework deals with the objectives to be accomplished 
within the DSS. Are the goals mostly tangible goals (e.g., time savings, cost savings, 
profit) or are they mostly intangibles (e.g., better decisions, more alternatives, user 
satisfaction, improved service for the customer)? 

The key to answering these questions is in knowing the organization's mission 
or goals. If an organization's goal is strictly increased profit, this goal is relatively 
easy to quantify. However, while increased profit is the primary goal, many or- 
ganizations have secondary goals. If the organization wants increased profits with 
increased organization morale, the goals are not as easily quantifiable. 

When answering the Level 2 question, the adjective “mostly” should be con- 
sidered. 

If the organization’s primary goals are mostly tangible with some intangible 
secondary goals, then it should take the the “tangible route.” If the organization’s 
primary goals are mostly intangible with tangible secondary goals, then the “intan- 
gible route” should be followed. 

In the case where the organization’s goals are unknown or the user cannot 
decide between tangible and intangible, the route label “both or unknown” should 


be chosen. However, this route will require the user to do a thorough analysis of 
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many methods to justify the DSS in order to find the method that best fits the 
organization. 

Many DSS are geared toward functional areas and may be far removed from 
the primary goals of an organization. In this case, the functional area goals need to 


be addressed in determining the most appropriate route in the framework. 


D. LEVEL 3 

The third level of the proposed framework leads the user to a method or a 
set of available methods considered to be the most appropriate to the user’s given 
sıtuation. 

The following routes are available for DSS Justification. 

1. Justifying/Tangibles 

If a user is trying to justify purchasing or developing a DSS and the 

benefits are mostly tangible, the third question is “Do intangible benefits have any 
impact on goals?” If the intangible benefits do have some impact, then the “yes” 


route will lead to the following methods of justification: 
e Excess Tangible Cost [Litecky, 1981] 
e Value Analysis [Keen, 1981] 


If the intangible benefits have little or no impact on goals, the “no” route 


will lead to the following methods: 
e Profitability Index [Garrison, 1988] 
e Cost Benefit Ratio [NDAC, 1980] 
e Savings Investment Ratio [NDAC, 1980] 
e [ncremental Analysis [Smith, 1983] 
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2. Justifying/Intangibles 
If the user is trying to justify a DSS and the goals are mostly 
intangible, the question to be addressed will be: “Is there a need to 
quantify intangible benefits?" An organization may desire a DSS that provides in- 
formation for better decisions (intangible), however, it wants to show a “bottom line 
figure” (tangible). The following methods are available to help give the organization 


that “bottom line figure”: 
e Excess Tangible Costs [Litecky, 1981] 
e Aggregate Benefit Value (weighted methods) [NDAC, 1980] 


If the organization does not require a bottom line figure, the following 


methods are more appropriate: 


e Value Analysis [Keen, 1981] 


e Benefit Profile Chart [Smith, 1983] 


Intuition [Hogue and Watson, 1983] 


e Success Stories 


These methods are discussed in Chapter II. 
3. Evaluate/Tangible 
When evaluating a DSS designed for mostly tangible benefits with some 
intangible impact on goals, Return on Investment in conjunction with User Satis- 
faction or Value Analysis would be appropriate. 
If the intangible benefits do not affect goals, simply Return on Investment 


(Actual) or Profit would be better measures. 
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4. Evaluating/Intangibles 
If an organization is evaluating a DSS designed for intangible benefits and 
there is still a need to quantify the answer, then the Expected Value Method would 
be appropriate. 
If there is not a need to quantify an evaluation, the following methods 


would be more appropriate: 


e User Satisfaction [Baroudi and Orlikowski, 1988] 


e Value Analysis [Keen, 1981] 


E. SUMMARY 

Organizations may have any number of goals. An organization may have a 
goal that is tangible in nature but the user desires a DSS that provides intangible 
benefits (e.g., more alternatives to help produce better decisions). The objective of 
the DSS now falls into the realm of intangibles even though the organization’s goals 
are tangible. To successfully use the decision tree, both the goals of the organization 
and the proposed purpose of the DSS must be taken into account and the appropriate 
measures applied as listed above. 

What follows are three scenarios which demonstrate the usefulness of the pro- 


posed framework. 
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F. SCENARIO 1 
The Commanding Officer for the U.S. Army Recruiting Command knows that 


a decision support system can help him with several decision making tasks such as: 


e From where are the majority of recruits for the U.S. Army 


coming from? 


e What can be done to increase recruitment in each geographic 


area? 


e What effect will increased advertising have in certain 


regions? 
e How much should be spent on advertising? 


The commander would like to procure a DSS, but must justify the 

expenditure to his superiors. Goals for the DSS are to increase recruitment 
and to increase the overall quality of individuals that the U.S. Army recruits. 

In this scenario the best response would be: 
Level 1. Justification 

The organization would like to procure a DSS. The DSS will require justifica- 
tion. 
Level 2. Tangible 

The scenarios goals are for increased recruitment and increased quality of 
recruits. These goals are tangible and can be measured. The recruiting command 
administers a aptitude test to judge overall quality of the recruit. These goals can 


be measured using traditional Cost Benefit Analysis techniques. 
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Level 3. 

The answer to the level three question is “No”, intangibles benefits do not 
impact on goals. 

The scenario does not state any intangible benefits as either primary or sec- 
ondary goals. 

This information leads the analysis to the following applicable alternative jus- 


tification methods: 


e [ncremental Analysis [Smith, 1983] 
e Profitability Index [Garrison, 1988] 
e Cost Benefit Ratio [NDAC, 1980] 


e Savings/Investment Ratio [NDAC, 1980] 


AN 


G. SCENARIO 2 


The human resource manager controls pay and suggests appropriate increases 
for each position within the organization. He would like a DSS to assist in the 
decision-making process. The DSS would be a great timesaver, but would primarily 
be used to satisfy the organization’s members and help keep morale at a high level. 

The DSS would have to be able to assist in dividing up the budget dollar for 
annual pay raises on a semi-annual and annual basis. Keeping employees happy is 
an important goal since to alienate them would contribute to the loss of outstanding 
employees. 

The DSS should be able to capture market information on what other organi- 
zations are paying (this can be accomplished through a financial service). 

The DSS should be able to incorporate the consumer price index. It should be 
able to show where the organization stands against the market on current salaries. 

It should be able to answer questions such as "What if I want to increase all 
programmers' salaries in my organization to account for the difference in their pay 
and the market leader? How much would this increase costs to the organization?" 

Top management would like the tangible benefits quantified and a list of all of 
the intangible benefits that the DSS will provide, but there is not a need to attempt 
to quantify improved morale and other intangible benefits. 

In this scenario, the best response would be: 

Level 1. Justification 

The organization would like to procure a DSS which requires justification. 
Level 2. Intangible 

The DSS would be used primarily to assist in the decision making process, to 
save valued time and to keep organization’s members motivated. The scenario lists 


several intangible goals but makes no mention of intangible goals. 
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Level 3. 

The answer to the level three question is ^No" there is not a need to quantify 
the intangibles benefits). 

Top management would like the tangible benefits quantified, but the intangible 
benefits should be listed. There is no need to place a specific dollar value to the 
intangible benefits. 

This information leads the human resource manager to these applicable alter- 


native justification methods: 


e Value Analysis [Keen, 1981] 
e Benefit Profile Chart [Smith, 1983] 


e Intuition [Hogue and Watson, 1983] 
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H. SCENARIO 3 

A major airline is currently running a decision support system for assisting 
allocation of planes to different flights and scheduling passengers for each flight. 
The DSS assists the decision-maker in insuring that planes are available for every 
flight. If a plane breaks down, the DSS can find another available plane to meet the 
grounded plane's commitments. 

The primary goal of the DSS is to increase the airline's profits through in- 
creased customer satisfaction. The better the organization's ability to meet all of 
its commitments, the more satisfied the customers should be. The assumption is 
made that when an airline has to cancel flights because of breakdowns, customers 
will begin to switch to a more reliable carrier. 

Management wants to know if an upgrade of the present system is needed. 
While the DSS in place is doing a satisfactory job, there is a question as to whether 
a newer system would be better. 

In this scenario, the best response would be: 

Level 1. Evaluation 

In the scenario, the airline presently has a DSS and would like to evaluate the 
DSS to assist the airline in making decision on whether to invest more money on 
upgrading their system. 

Level 2. Tangible 

The goals in this scenario are centered around increasing the airlines profits. 
Level 3. 

Yes (Intangibles Benefits have an impact on goals) 

The primary goal is to increase profits. However, customer satisfaction does 


have an impact on the goals of increased profits. 
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This information leads the organization to the following analysis: 


e ROI w/Value Analysis 


e ROI w/User Satisfaction 
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V. CONCLUSION 


A. THERE IS A NEED FOR DSS 

Decision Support Systems can play a vital link in the area of decision making. 
DSS are not intended to make final decisions, rather to assist the decision maker in 
the decision making process. Expanding a decision makers views and alternatives, 
combined with the ability to run sensitivity analysis on each alternative, can greatly 
enhance the quality of a decision. However, not every DSS is successful. There must 
be some means set in place to justify and evaluate a DSS. Even if the evaluation 
is nothing more than a user satisfaction survey, this is still a method which will 
provide feedback to top-level management. Without some sort of feedback method, 
the organization will simply run blind. 

In Chapter III, several methods of justification and evaluation 


are presented. Each method has advantages and disadvantages. 


B. COST BENEFIT METHODS 
“Cost /benefit evaluations will probably provide the most meaningfu: results to 
management.” [Sprague and Carlson, 1982] However, CBA runs into trouble when 


trying to evaluate or justify intangible benefits. 


C. VALUE ANALYSIS 

This method is simple to implement, using prototyping which can help limit 
the initial investment while testing the water to see if the system will be a success. 
However, it is “less rigorous than cost benefit techniques and may not include all 


the measures that are relevant.” [Sprague and Carlson, 1982] 
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D. USER ATTITUDE/SATISFACTION 


Using surveys to measure user satisfaction can also run into problem areas. 
[Sprague and Carlson, 1982] lists the following problem areas when using question- 


naires: 


1. In developing a questionnaire to measure attitudes, one may be trying to 
quantify that which is not quantifiable and the subsequent analysis may be 


misleading. 


2. Questionnaire respondents may interpret questions differently than intended, 


and answers to some questions may influence others. 


3. Administering questionnaires may be an inconvenience to individuals and ex- 
pensive (in terms of hours lost by the respondents and hours spent by the 


interviewers). 


4. The method does not identify the causes of any measured changes. 


E. OTHER METHODS 


e Expected Value 

e Incremental Analysis 

e Excess Tangible Costs 

e Aggregate Benefit Value 


Depending on the organization and its goals, these methods can be very ben- 
eficial and simple to use. However, they tend to rely on subjective judgement when 
establishing weighting or probability of occurrence figures. In some cases, the slight- 
est change in weights or probabilities can drastically change the outcome of the 


evaluation. 


4] 


F. USE OF THE DECISION TREE 

The main purpose of this thesis was to provide a framework that can be used to 
assist the decision maker in determining the most appropriate method for evaluating 
or justifying a DSS. The decision tree that is provided in Chapter IV (along with a 
description of the methods) allows a decision maker to quickly assess the methods 
that best suits his personal or organizational needs. The methods for evaluating 
and justifying a DSS will continually change. The decision tree can be incorporated 
to handle any changes. The basic premise for the decision tree is based on three 


questions that the organization must answer: 


l. Is the decision maker attempting to justify future procurement of a DSS or is 


the decision maker attempting to evaluate the current DSS? 


2. Are the goals of the organization mostly tangible or intangible? If the DSS is a 
personal DSS for an individual, then the individual’s goals should be analyzed. 
Hopefully, the individual decision makers goals complement the organizational 


goals. 


3. If the goals are mostly tangible, do the intangible benefits have any impact on 


obtaining the organizational goals? 


4. If the goals are mostly intangible, is there a need to quantify intangible bene- 


fits? 


By successfully answering these questions, the decision tree should lead the 
decision maker to a method(s) that will provide the best available results given the 


organization’s views and goals and prove acceptable to top-level management. 
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G. SIX POINTS 


While attempting to justify or evaluate a DSS, the following points should be 


kept in mind: 


1. DSS can provide both tangible and intangible benefits. All appropriate bene- 


fits should be considered when the justification process begins. 
2. Benefits should be viewed in light of the organizational objectives. 


3. The use of a personal DSS should be viewed in light of the user's objectives, 


and should coincide with that of the organization. 


4. Evaluation of a DSS is an important feedback mechanism that helps to provide 


information to top-level management. 


5. Evaluations throughout the lifecycle of the DSS are necessary to minimize the 


risks of continually pouring money into an unsuccessful system. 


6. Doing one or two things well is far better than doing several things that are 
mediocre. Do not try to provide a DSS that delivers everything because it can 


truly cost everything. 
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APPENDIX A 
Cost Benefit Profile Chart [20] 


Intangible benefits of MIS/DSS leading to improved business performance. 


e Category A = Some improvement in existing system. 
e Category B = Significant improvement in existing system. 


e Category C = Significant new benefit. 
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Business Function/A ctivity Degree of Performance Improvement 


Interactive Problem Solving within 
Company 

Interactive Problem Solving with 
Customer 

Stimulation of new ideas 

(i.e., graphics) 

Faster Design (i.e., computer-aided 
design) 


Processing of engineering change 
orders 


Management of Professional's time 
e.g., reduced clerical workload 
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Business Function/Activity Degree of Performance Improvement 
Finance/Accounting 
Budget Preparation 
Use of operating/leverage 
Privacy of data information 
Security of data information 


Integrity (accuracy of data) 


EE ME M ae 
Capital Budgeting 7| d 
Auditing and Internal Control a 
Timely Reports Er 
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Business Function/Activity Degree of Performance Improvement 


Employee Relations/Human Resources elle 
Identification of best performers 
(Individual and Groups) 


Standards 

Higher motivation for workforce EX 
— Career planning capabilities DS RR 
— Turnover/absenteeism Pa o 
— Job satisfaction CNN 


Employee Training 
e.g., computer assisted instructions 
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Business Function/Activity Degree of Performance Improvement 


Employee Relations/Human Resources ENEMIES 
Understanding of how HRM functions a lll 
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Business Function/Activity Degree of Performance Improvement 


General Management 


Increased Communications among 
departments 


assessible 
Aal ee eee eee ers. 
Easier to use system d 


Increased secretarial efficiency, 
effectiveness 


Improved Accuracy/Conciseness 
Timeliness relevance of all 
information 
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Business Function/Activity Degree of Performance Improvement 


General Management leie 
— Need fewer computer programs Do m 
— Need less program maintenance App 
— Reduced travel costs p LL dm 
— Better use of programmers time | Clo le 


Business Function/ Activity 


Decision Support System alee 
— Goal seeking |] dX 3 


The Cost Benefit profile is not meant to fit every aspect of a particular organization. It 
must be tailored to a specific organication. This is strictly a guideline of a Cost Benefit 
Profile. 
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APPENDIX B 


Project Year Discount Factors 


TABLE A TABLE B 
Present Value of $1 (Single Present Value of $1 (Cumulative 
Amount: Used when cash flows Uniform Series: To be used 
accrue in varying amounts when cash flows accrue in the 


each year) same amount each year) 
Project 10% 10% 
1 0.954 0.954 
2 0.867 1.821 
3 0.788 2.609 
4 0.717 3.326 
3 0.652 3.977 
6 0.592 4.570 
7 0.538 5.108 
8 0.489 5.097 
9 0.445 6.042 
10 0.405 6.447 
ul 0.368 6.815 
12 0.334 7.149 
13 0.304 1.453 
14 0.276 1.729 
15 0.251 1.980 


Note: Table B factors represent the cumulative sum of Table A factors through any 


given project year. 


Reference: [NDAC, 1980] 
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APPENDIX C 


User Satisfaction Survey 


This appendix provides a basic user satisfaction survey [Baroudi and Or- 
likowski, 1988]. The survey is not intended to meet every organizations needs. 
However, it does provide the basis for a user satisfaction survey, which should be 
tailored to an organization's needs and goals. 

This survey should be administered to the users of the DSS after weights have 
been placed on each of the possible responses. Likert scale normal runs from 1 
to 7 points. It is up to the organization to establish an acceptable score prior to 
administering the survey. 


Please follow these instructions: 


1. Check each scale in the position that describes your evaluation of the factor 


being described. 
2. Check each scale, do not omit any. 
3. Check only one position for each scale. 


4. Check in space, not between spaces. 


ANSWER BASED ON YOUR OWN FEELINGS 
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l. Relationship with the Management Information Department (MID) staff 


dissonant : : : : : : : : harmonius 


bad E : : : good 


2. Processing of requests for changes to existing DSS 


fast : : : : : : : : slow 


untimely timely 


Ki . . > 
a= =p um o * em em em wem em em *° em em em wem em wem `" em em em * 


3. Degree of training provided to users 


complete incomplete 


mm ’ mn * mm ° m mm ° mm m O) m mm ° mm m 


low euren 


4. User understanding of the DSS 


insufficient : : : : : : : : insufficient 


=_ =» = --— => =» = > = =a => = * - mm = «=p =» -— e = 


complete incomplete 


* - mm e * amu * wem em em * ao o tum * mn m ° an m ° 
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9. User's feeling of participation 


positive negative 


insufficient sufficient 


. 
= ND ap -——— em * em em em `" em em wm ° u um 7 o o o ° oos 


6. Attitude of the Management Information Department staff 


cooperative tt sie ee Zenon 


negative positive 


Z em em em `" em em ° um m ° em wm em ° um m ° m nm ° mm m 


7. Reliability of output information 


high Veen ir menge Be ow 


superior inferior 


% un °? aw aw aw °? aw æn m ° aw an an ° o w æ ° aw ao ao °? wae ° 


8. Relevancy of output information (Does the DSS do intended function) 


useful : : : : : : : useless 


relevant : : : : : : í : irrelevant 
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D 


10. 


SE 


12. 


Accuracy of output information 


inaccurate BENNETT ote accurate 
low ERE en high 
Precision of output information 

low a ed 
definite : : : : : : : uncertain 


Communication with the Management Information Department staff 


dissonant : : : : : : : : harmonious 


ma a * am m m = mm ° a m m ° em em em a= e— mm -— a mm 


destructive productive 


“aaa C em em em `" em em em ° mm m ° m mm ° mm m ° em em em * 


Time required for the new systems development 


unreasonable : : : : : : : : reasonable 


m zw = = == em = = -- GNE = = m A e = a= ex = A m 


acceptable unacceptable 


D em em em ° m m m " m mm ° mm m ° ew m m ° mm m ° u am am * 
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I3: 


14. 


19. 


16. 


bi 


Completeness of the output information 


sufficient 


adequate 


DSS assisted in decision making process 


definite o LE A ee 


superior See: Ve ar: 


Is DSS easy to learn and use 


easy A EE oc d" 


fast 


wem wem wem  Ë wem wem em ° em em em ° am m ° ann ° un m ° mn em 


List the functions that are the most beneficial: 


List the functions that are most needed: 
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. o . . *. * *. *. 
ÅÅ- > = =p =» == uu =» - == =» = == =» -—- A— =» o E 4» 


. -— rn m tm s = ap m tn m y -æ m * = =» =» tm 


ınsufficient 


inadequate 


uncertain 


inferior 


difficult 


slow 


10. 


nr. 


12. 
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